# Relative position encoding

BEiT is a Vision Transformer-based model pre-trained on ImageNet-21k through self-supervised learning and fine-tuned on ImageNet-1k for image classification tasks.

BEiT is an image classification model based on Vision Transformer (ViT) architecture, pretrained with self-supervised learning on ImageNet-21k and fine-tuned on ImageNet-1k.

Transformer-XL is a causal Transformer architecture that uses relative position encoding. It can capture longer context by reusing previously computed hidden states, making it suitable for text generation tasks.